Image searching method and image processing method

ABSTRACT

The image searching method extracts or recognizes specific information for an image that exists in the image and/or accompanies it, allows either the specific information itself or related information thereof to be appendant to image data for the image, stores the image data as well as either information appendant to the image data as accessory information in a database, designates at least a portion of the accessory information as a searching condition, searches through the database under the designated searching condition, determines a degree of agreement with the accessory information appendant to a selected image data and reads an image having the degree of agreement not lower than a predetermined value. The image processing method utilizes the image searching method. These methods are capable of efficient image detection or search using as a key specific information such as the shape of subjects.

BACKGROUND OF THE INVENTION

[0001] This invention relates to an image searching method and an imageprocessing method. More specifically, this invention relates to an imagesearching method in which specific information extracted from an imagesuch as simple figures or subjects such as human individuals or physicalobjects or specific information that accompanies the image such as audioinformation or message information such as characters and marks isstored in a database as accessory information for the image and imagesare searched through on the basis of the stored accessory information.

[0002] The present invention also relates to an image searching methodwhich extracts the subjects of interest on the basis of the positioninformation, camera information and map information that are relevant tothe time when the image was taken and which uses the result ofextraction (in the form of image) to search through images.

[0003] The invention also relates to an image processing method whichutilizes such image searching methods or the specific informationextracted from the image.

[0004] When viewing a portrait, we pay the most attention to the face ofthe person in the picture. Therefore, when the original image typicallyrecorded on a film is to be printed on a duplicating material such asprint paper, the amount of exposure has to be determined such that theface of the person is reproduced in an appropriate color and density.

[0005] To meet this requirement of portraiture, Unexamined PublishedJapanese Patent Application (kokai) No. 346332/1992 proposed a methodcomprising the steps of dividing an original color image into a multipleof pixels, separating each pixel into three colors R, G and B,performing photometry, constructing histograms about hue and saturationon the basis of the result of photometry, slicing as a human faceequivalent area those regions of the histograms which contain pixelshaving a hue and saturation that correspond to the face of the person inthe portrait, and determining the correct exposure on the basis of thephotometric data for the sliced regions.

[0006] In order to increase the probability for extracting the humanface equivalent area, Unexamined Published Japanese Patent Application(kokai) No. 160993/1994 proposed various techniques such as removing thearea contiguous to the edges of the image as the background region ortransforming the extracted area into a figure defined by lines anddetermining whether the extracted area corresponds to the face of theperson in the portrait on the basis of the shapes of neighboring areassurrounding the extracted area and the shape of the extracted area.

[0007] These techniques, however, have a common problem in that if theoriginal image contains the ground, trees and any other areas having ahue and saturation close to those of flesh color, such areas may beerroneously interpreted as the human face equivalent area. If theseareas are adjacent the true human face equivalent area, they cannot becorrectly separated from the latter, with the result that the originalimage is not likely to be divided into areas of appropriate sizes.

[0008] If the areas other than the face of the person in the portraitare erroneously interpreted as the human face equivalent area or if theareas adjacent the human face equivalent area which have a similar hueand saturation cannot be separated from the latter, it has beenimpossible to determine the correct exposure for appropriate printing ofthe true human face.

[0009] To solve this problem, it would be effective to utilize not onlythe color (hue and saturation) of the area of interest but also theinformation about its shape. Speaking of the human face equivalent area,it will in most cases have an elliptical or oval shape, so this specialshape may be added to the color information as a condition forextracting the human face equivalent area.

[0010] Thus, the information about the shapes of subjects in theoriginal image is very important to the purpose of image detection orsearching. In this connection, reference may be had to the technologydisclosed in commonly assigned Unexamined Published Japanese PatentApplication (kokai) No. 138471/1997. In this technology, the edges of animage are detected and the lines of contours of an area corresponding tothe person in portrait are extracted so that the area, particularly theone corresponding to the face of the person, is rendered to reproducethe preferred color.

[0011] According to another prior art image searching system, the imagethe searcher wants to extract can be searched for without requiring thesearcher to draw the image of interest or even if no image close to theone wanted by the searcher is ready at hand or within the list ofavailable images. For this image searching system, reference may be hadto Unexamined Published Japanese Patent Application (kokai) Nos.21198/1995 and 249352/1996.

[0012] For use in taking documentary pictures by laymen, cameras haverecently been put on the commercial market that provide automaticrecording of “when and where” the picture was taken. Those cameras are acomposite product in which an artificial satellite based globalpositioning system (GPS) commonly incorporated in car navigation systemsis combined with a direction sensor and a camera [Shashin Kogyo(Photographic Industry), 1996-7, pp. 12-15, 84-87).

[0013] While photography has several roles to play, the importance ofphoto's documentary and evidential performance is one of the essentialvalues of photograph. This is why accessory information telling “whenand where” the subject was shot is important to documentary pictures.Conventional cameras have so far provided the information about “when”to some extent but not the information about “when”.

[0014] In this sense, the camera described as a new product in ShashinKogyo, supra (which is hereunder sometimes referred to as a GPSincorporating camera) may well be considered a device that can fulfilthe inherent functions of documentary pictures. First of all, the datingfunction of the conventional “dating” camera and the representation oftime by the GPS incorporating camera are entirely different in terms ofevidential value: the former provides time that can be determined“arbitrarily” (for personal use) whereas the latter records the“official time” being sent from the satellite.

[0015] Exactly speaking, the position information provided by the GPSincorporating camera does not tell the position of the subject butindicates the position of the camera and the direction of shooting.Therefore, if the picture is a long shot, the position of the subject isnot always definite. Even in this case and if there is a need, theposition of the subject can be correctly calculated from, for example,the distance between that subject and another subject in the image.

[0016] The present inventors previously proposed in Japanese PatentApplication No. 162130/1998 a photographic printing method comprisingthe steps of acquiring at least one kind of shooting information inassociation with the image taken that is selected from among shootingposition, shooting direction and magnification rate, obtainingadditional information associated with at least a portion of theacquired shooting information and adding the obtained additionalinformation to a photographic print. The present inventors also proposeda photographic printing apparatus implementing the method.

[0017] Using the photographic printing method and apparatus, thecustomer looking at the picture can be easily reminded of relevantatmosphere such as the circumstances of shooting on the basis of theshooting information added in association with the recorded picture.Thus, the method and apparatus can not only enhance the documentaryfunction of photographic prints but also allow them to serve the purposeof amusement (by furnishing a topic of conversation).

SUMMARY OF THE INVENTION

[0018] The present invention is a further development of the concept ofthe above-described prior art for detecting areas of interest usingspecific shapes in the original image as a key. It is therefore thefirst object of the invention to provide an image searching methodcapable of efficient image detection or search (which is hereundercollectively referred to as image search) using as a key the shapes ofsubjects such as a person or a physical object in the original image orspecific information that accompanies the original image such as audioinformation or message information.

[0019] The present invention has been accomplished with a view tofurther facilitating the use of this technology. It is therefore thesecond object of the invention to provide an image searching method thatis applicable when the customer wants to locate a desired image from anumber of stored images and which enables easy image search by utilizingthe position information, camera information and map information thatare relevant to the time when the image was taken and which is added inassociation with the recorded image.

[0020] The third object of the invention is to provide an imageprocessing method which utilizes such image searching methods or thespecific information extracted from the original image.

[0021] In order to attain the above overall object, the presentinvention provides an image searching method, comprising the steps of:

[0022] extracting or recognizing specific information for an image thateither exists in the image or accompanies it or both exists in the imageand accompanies it; allowing either the specific information as-obtainedor related information relative to the specific information to beappendant to image data for the image; storing the image data as well aseither information appendant to the image data as accessory informationin a database; designating at least a portion of the accessoryinformation as a searching condition; searching through the databaseunder the designated searching condition; determining a degree ofagreement with the accessory information appendant to a selected imagedata; and reading an image having the degree of agreement not lower thana predetermined value.

[0023] In order to attain the first object, the present inventionprovides an image searching method, comprising the steps of: applying toan image a treatment of extracting M (M≧1) kinds of specific geometricfigures predetermined for the image; allowing to be appendant to imagedata of the corresponding image an accessory information containing acombination of position and size or a combination of position, size anddirection or a combination of position, size and color or a combinationof position, size, direction and color that correspond to N (N≧0)specific geometric figures that have been obtained as a result ofextraction; storing the image data as well as the accessory informationappendant to the image data in a database; designating, for L (L is fromone to M) kinds of the specific geometric figures, presence or absenceof the specific geometric figures or an outline of at least one elementselected from among the position, size, direction and color; determininga degree of agreement with the accessory information appendant to imagedata of the stored image; and reading an image having the degree ofagreement not lower than a predetermined value.

[0024] Preferably, the specific information existing in the image is M(M≧1) kinds of specific geometric figures predetermined for the image,the related information to be stored as the accessory information iseither a combination of position and size or a combination of position,size and direction or a combination of position, size and color or acombination of position, size, direction and color that correspond to N(N≧0) specific geometric figures that have been obtained as a result ofextraction by processing the image with a treatment of extracting the Mkinds of the specific geometric figures, and the designating step of atleast a portion of the accessory information as the searching conditioncomprises designating, for L (L is from one to M) kinds of the specificgeometric figures, presence or absence of the specific geometric figuresor an outline of at least one element selected from among the position,size, direction and color.

[0025] Preferably, the specific information existing in the image is M(M≧1) kinds of specific subjects predetermined for the image, therelated information to be stored as the accessory information is eithera combination of position and size or a combination of position, sizeand direction or a combination of position, size and color or acombination of position, size, direction and color that correspond to N(N≧0) specific subjects that have been obtained as a result ofextraction by processing the image with a treatment of extracting the Mkinds of the specific subjects, and the step of designating at least aportion of the accessory information as the searching conditioncomprises designating, for L (L is from one to M) kinds of the specificsubjects, presence or absence of the specific subjects or an outline ofat least one element selected from among the position, size, directionand color.

[0026] Preferably, the image is temporally continuous images, thespecific information existing in the image is a specific geometricfigure predetermined for the image, the related information to be storedas the accessory information comprises information about the specificgeometric figure and a direction of movement of an image correspondingto the specific geometric figure that have been obtained by processingthe temporally continuous images by a treatment of extracting thespecific geometric figure from the temporally continuous images atpredetermined intervals of time or frame numbers and determining thedirection of movement of the image corresponding to the geometric figurebased on comparison of a position of the extracted specific geometricfigure between frames, and the step of designating at least a portion ofthe accessory information as the searching condition comprisesdesignating, for the temporally continuous images, outlines of thespecific geometric figure and the direction of its motion vector at thepredetermined intervals of time or frame numbers.

[0027] Preferably, the image is temporally continuous images, thespecific information existing in the image is a specific subjectpredetermined for the image, the related information to be stored as theaccessory information comprises information about the specific subjectand a direction of movement of an image corresponding to the specificsubject that have been obtained by processing the temporally continuousimages by a treatment of extracting the specific subject from thetemporally continuous images at predetermined intervals of time or framenumbers and determining the direction of movement of the imagecorresponding to the specific subject based on comparison of a positionof the extracted subject between frames, and the step of designating atleast a portion of the accessory information as the searching conditioncomprises designating, for the temporally continuous images, outlines ofthe specific subject and the direction of its motion vector at thepredetermined intervals of time or frame numbers.

[0028] Preferably, the image has audio information, the specificinformation accompanying the image is the audio information, theaccessory information is the audio information that has been recognizedfor the image having the audio information, the step of designating atleast a portion of the accessory information as the searching conditioncomprises designating an outline of audio information for the imagehaving the audio information, and the degree of agreement is with speechrecognition information that is appendant to the stored image data.

[0029] Preferably, the accessory information comprises not only theaudio information but also the related information which is either acombination of position and size or a combination of position, size anddirection or a combination of position, size and color or a combinationof position, size, direction and color which correspond to a specificgeometric figure, and the searching condition comprises not only theaudio information but also presence or absence of the specific geometricfigure or at least one element selected from among position, size,direction and color which correspond to the specific geometric figure.

[0030] Preferably, the accessory information comprises not only theaudio information but also the related information which is either acombination of position and size or a combination of position, size anddirection or a combination of position, size and color or a combinationof position, size, direction and color which correspond to a specificsubject, and the searching condition comprises not only the audioinformation but also presence or absence of the specific subject or atleast one element selected from among position, size, direction andcolor which correspond to the specific subject.

[0031] Preferably, the specific information which both exists in andaccompanies the image is person's information comprising a facial imageof a particular person and his or her name, the related information tobe stored as the accessory information is presence or absence of theparticular person or either a combination of position and size of theparticular person or a combination of position, size and direction ofthe particular person or a combination of position, size and color ofthe particular person or a combination of position, size, direction andcolor of the particular person and obtained by the steps ofpreliminarily registering person's information comprising the facialimages of the A particular persons (A≧1) and their names and performingextraction of persons and a person identifying treatment using theperson's information at the time of storing the image, the step ofdesignating at least a portion of the accessory information as thesearching condition comprising designating presence or absence of Bparticular persons (B is from one to A) or at least one element selectedfrom among positions, sizes and directions of the particular persons,and the degree of agreement is with the accessory information to each ofthe stored images.

[0032] Preferably, the image is temporally continuous images, and therelated information to be stored as the accessory information isobtained by processing the temporally continuous images by the personidentifying treatment using the person's information at predeterminedintervals of time or frame numbers and comprises information about theparticular parsons and a direction of the movement of the particularparsons, the step of designating at least a portion of the accessoryinformation as the searching condition comprising designating presenceor absence of the particular persons or at least one element selectedfrom among positions, sizes and colors of the particular persons, andthe direction of its motion vector.

[0033] Preferably, the image has audio information, the accessoryinformation comprises the related information obtained for the imagehaving the audio information by a person identifying treatment using theperson's information and audio information recognized for the imagehaving the audio information, and the step of designating at least aportion of the accessory information as the searching conditioncomprises designating an outline of the related information containingthe presence or absence of the particular persons or at least oneelement selected from among positions, sizes, directions and colors orthe audio information as the searching condition applicable to the imagehaving the audio information.

[0034] Preferably, the image has message information, the specificinformation accompanying the image is the message information, theaccessory information is the message information recognized for theimage having the message information, the step of designating at least aportion of the accessory information as the searching conditioncomprises designating an outline of message information for the imagehaving the message information, and the degree of agreement is withmessage recognition information appendant to the stored image data.

[0035] Preferably, the message information is text data or appendantimage data.

[0036] Preferably, the image is a computer graphics image, the specificinformation appendant to the image is editing data used to create theimage, the accessory information is the editing data, the step ofdesignating at least a portion of the accessory information as thesearching condition has the substeps of designating a scene of thecomputer graphics image by at least one element selected from amongposition, size and direction of at least one member selected from amonga specific geometric figure, a specific physical object and a specificcolor and automatically transforming the designated scene designationdata to a computer graphics command, and the step of determining thedegree of agreement is for determining the degree of agreement betweenthe resulting computer graphics command and the editing data appendantto a selected item of the stored image data.

[0037] Preferably, the computer graphics image is temporally continuouscomputer graphics images, the accessory information comprises not onlythe editing data but also either the specific geometric figure or thespecific physical object or both and a direction of movement of an imagethat have been obtained by determining, for the temporally continuouscomputer graphics images, the direction of movement of the imagecorresponding to either the specific geometric figure or the physicalobject or both at predetermined intervals of time or frame numbers basedon comparison between frames of the position of either the specificgeometric figure or the specific physical object or both, and outlinesof either the specific geometric figure or the specific physical objector both and the direction of their motion vector are designated as thesearching condition at predetermined intervals of time or frame numbersfor the temporally continuous computer graphics images.

[0038] Preferably, the step of designation has the substeps ofdesignating at least a portion of the accessory information by at leastone means selected from among symbols, language and speech andautomatically transforming at least one designated element selected fromamong symbols, language and speech into the specific information oraccessory information therefor.

[0039] Preferably, an area in the image data that meets the searchingcondition is displayed in temporal preference.

[0040] Preferably, an area in the image data that meets the searchingcondition is displayed as enhanced by a different image processingscheme than applied to the other areas.

[0041] Preferably, the image data is processed in accordance with adisplay performance of a display terminal.

[0042] Preferably, the image areas that have been searched through aremanaged area by area and a priority of reference in search mode is setin accordance with access frequency.

[0043] Preferably, the image areas that have been searched through aremanaged area by area and an image resolution for data storage is set inaccordance with access frequency.

[0044] In order to attain the second object, the present inventionprovides an image searching method, comprising the steps of; extractinga name of a shot place where an image was shot or a subject and theirposition on a map; storing image data of the image as well as anextracting result allowed to be appendant to the image data as accessoryinformation: designating the position or the name of the shot place orthe position or the name of the subject as reference data; determining adegree of agreement with the accessory information appendant to imagedata of the stored image; and

[0045] reading an image having the degree of agreement not lower than apredetermined value.

[0046] Preferably, the specific information accompanying the image is aname of a shot place where an image was shot or a subject and theirposition on a map, the accessory information is a result obtained byextracting the name of the shot place or the subject and theirs positionon the map from shooting information when the image was shot and mapinformation, and the step of designating at least a portion of theaccessory information as the searching condition comprises designatingthe position or the name of the shot place or the position or the nameof the subject as reference data.

[0047] Preferably, the shooting information is acquired in associationwith the shot image and is either a first data composition consisting ofshooting position or a second data composition consisting of shootingposition and shooting direction or a third data composition consistingof shooting position and magnification rate or a fourth data compositionconsisting of shooting position, shooting direction and magnifying powerof shooting.

[0048] Preferably, an outline of depth is designated as reference dataconcerning the shooting position and subject's position by arrangingpatterned subjects in a virtual map space.

[0049] Preferably, the shot place or the position and size of thesubject within the screen are extracted and added to the accessoryinformation together with the extracted information.

[0050] Preferably, the shot place or the position and size of thesubject within the screen are extracted and added to the designatedinformation together with the extracted information.

[0051] Preferably, a difference between image data of the actual shotimage and image data simulating the shot image and which has beencreated from position information and camera information when the imagewas shot, and the map information is stored and when the image which isa result of searching is outputted, the stored difference is added tothe image data simulating the shot image and which has been created fromthe position information, the camera information and the map informationand the resulting image is output.

[0052] The present invention provides an image searching method,comprising the steps of: deriving information representing a distancefrom a screen recording an image; storing image data of the image aswell as a deriving result allowed to be appendant to the image data ofthe image as accessory information; designating the informationrepresenting the distance from the image recording screen as referencedata; determining a degree of agreement with the accessory informationappendant to image data of the image; and reading an image having thedegree of agreement not lower than a predetermined value.

[0053] Preferably, the specific information accompanying the image isinformation representing a distance from a screen recording the image,the accessory information is a result obtained by deriving theinformation representing the distance from the image recording screen,and the step of designating at least a portion of the accessoryinformation as the searching condition comprises designating theinformation representing the distance from the image recording screen asreference data.

[0054] Preferably, if the image is a multi-stage focused image or astereoscopic image, the information representing the distance from theimage recording screen is derived from the multi-stage focused image orstereoscopic image.

[0055] Preferably, the information representing the distance from theimage recording screen is derived from the position information when theimage was shot and map information or from the position information, themap information and the camera information.

[0056] Preferably, an outline of the depth for each of the areas in theimage recording screen is designated as reference data concerning theinformation representing the distance from the image recording screen.

[0057] Preferably, an outline of the depth is designated as referencedata concerning the information representing the distance from the imagerecording screen by arranging patterns of subjects in a virtual mapspace.

[0058] Preferably, a difference between image data the actual recordedimage and image data simulating the recorded image and which has beencreated from position information, camera information and mapinformation that are relevant to a shooting time is stored and to outputthe image which is a result of searching, the stored difference is addedto the image data simulating the recorded image and which has beencreated from the position information, camera information and mapinformation that are relevant to the shooting time and the resultingimage is output.

[0059] In order to attain the third aspect, the present inventionprovides an image processing method, comprising the steps of: extractingspecific figures or specific subjects in a printed image during aprevious printing; storing the extracted specific figures or subjects ina database as accessory information that is keyed to image processingconditions either alone or in combination with the image data; readingimage data from a previous print or captured image data recording mediumto extract the specific figures or specific subjects for processingprints upon reorder; determining a degree of agreement between theextracted specific figures or subjects and the accessory informationstored in the database; searching for an image having the degree ofagreement not lower than a predetermined value; and processing a hitimage under the image processing conditions employed for the previousprinting.

[0060] The present invention provides an image processing method,comprising the steps of: storing image processing conditions for aprevious printing in a database in such a manner that they are keyed toimage data; reading image data from a previous print or captured imagedata reading medium to extract a first specific figure or subject forprocessing prints upon reorder; also extracting a second specific figureor subject from the image data stored in the database; determining adegree of agreement between the first specific figure or subject and thesecond specific figure or subject; searching for an image having thedegree of agreement not lower than a predetermined value; reading fromthe database the image data for a hit image together with the storedimage processing conditions keyed to the image data; and subjecting thethus read image data to image processing under the image processingconditions as read from the data base.

[0061] The present invention provides an image processing method,comprising the steps of: extracting specific figures or specificsubjects from a plurality of images within a single order; andperforming image processing of image frames to yield prints that arefinished to have quality within a predetermined range if the imageframes are similar in terms of positions and sizes of the specificfigures or subjects.

BRIEF DESCRIPTION OF THE DRAWINGS

[0062]FIG. 1 shows an original image which is to be searched through byan image searching method according to an example of the invention;

[0063]FIG. 2 shows a designated searching pattern entered by the user tosearch through the original image shown in FIG. 1;

[0064]FIG. 3 shows the original image of FIG. 1 with extracted specificfigures superposed thereon;

[0065]FIG. 4 is a flow diagram showing the outline of steps in apreliminary process of extracting predetermined specific figures fromthe original image shown in FIG. 1;

[0066]FIG. 5 is a flow diagram showing the sequence of steps in theprocess of searching through images on the basis of the data stored inmemory means;

[0067]FIG. 6 shows a subject designated by an image searching methodaccording to another example of the invention;

[0068]FIG. 7 shows an example of the result of searching for the subjectin response to the designated shape in FIG. 6;

[0069]FIG. 8 shows a subject designated when dynamically changing imagesare searched through by an image searching method according to stillanother example of the invention;

[0070]FIG. 9 shows an example of the result of searching for dynamicallychanging images in response to the designated shape in FIG. 8;

[0071]FIG. 10 shows an original image 10 on film that was taken with aGPS incorporating camera and which is to be searched through by an imagesearching method according to yet another example of the invention;

[0072]FIG. 11 shows a designated searching pattern 20 entered by theuser to search through the original image shown in FIG. 10;

[0073]FIG. 12 is a flow diagram showing the outline of preliminary stepsfor an image searching method according to another example of theinvention that are performed to specify the positions and names ofsubjects whose positions in the original image 10 can be specified;

[0074]FIG. 13 is a flow diagram showing the sequence of steps in theprocess of searching through images on the basis of the data stored inmemory means by an image searching method according to a further exampleof the invention;

[0075]FIG. 14 is an exemplary designated searching pattern containingdepth information for use in still another example of the invention;

[0076]FIG. 15A is a diagram showing the relationship between positioninformation and the direction of viewpoint for the case where thesubject of interest can be identified;

[0077]FIG. 15B is a diagram showing the relationship between positioninformation and the direction of viewpoint for the case where theshooting place can be identified but the identity of the subject ofinterest cannot be identified; and

[0078]FIG. 16 is a block diagram showing an exemplary print system thatimplements the image processing method of the invention and that employsan image searching apparatus for implementing the image searching methodof the invention.

DETAILED DESCRIPTION OF THE INVENTION

[0079] The image searching method and image processing method of theinvention are described below in detail with reference to the preferredembodiments shown in the accompanying drawings.

[0080] First, by referring to FIGS. 1-9, we describe in detail the imagesearching method according to the first embodiment of the invention andthe image processing method according to the third embodiment of theinvention.

[0081]FIGS. 1-3 are each intended to illustrate the image searchingmethod according to an example of the first embodiment of the invention.FIG. 1 shows an original image 10 that is supplied from an image datasource (not shown) and the image data for which is to be stored inmemory means such as a database (not shown). FIG. 2 shows a designatedsearching pattern 20 which the user has designated with an imagedesignating device (not shown) by, for example, manual drawing on atablet typically with a pen or by inputting to an image input device orgraphically inputting to an image display device, typically by means ofa mouse or from a keyboard. FIG. 3 shows the result 30 of extractingpredetermined specific geometric figures (e.g. circles, triangles andrectangles which are hereunder referred to simply as specific figures)from the original image 10 in FIG. 1 by the program for the imagesearching method according to the example under consideration using asuitable technique such as pattern matching.

[0082] It is also possible to detect various specific shapes by othertechniques such as the edge tracing method disclosed in commonlyassigned Unexamined Published Japanese Patent Application (kokai) No.138471/1997, supra.

[0083] In the original image 10 shown in FIG. 1, numeral 11 represents atriangular mountain, 12 and 13 represent the face and the trunk,respectively, of a person, and 14 represents a building. Referring nowto the designated searching pattern 20 shown in FIG. 2, the user in anattempt at searching for the original image 10 shown in FIG. 1 hasarranged a triangle 21 symbolizing the mountain, two ellipses 22 and 23symbolizing the face and the trunk, respectively, of the person, and avertically elongating rectangle 24 symbolizing the building, with eachfigure being fitted in the position where it is most likely to be found.

[0084] In FIG. 3, the figures extracted from the original image 10 inFIG. 1 by the program for the image searching method according to theexample under consideration are indicated by dashed lines and superposedon the actual objects in the original image 10 that are indicated bysolid lines (for clarity, however, the dashed lines are slightly offsetfrom the solid lines). To be specific, kinked line 31 corresponds to thetriangular mountain 11 in the original image 10, circles 32 and 33correspond to the face 12 and trunk 13, respectively, of the sameperson, and rectangle 34 corresponds to the building 14. A rectangle 35bridging the windows on the building shown on the right side of FIG. 3and an ellipse 36 below the building have been erroneously detected bythe program.

[0085] The original image 10 to treated by the image searching method ofthe invention is not limited in any particular way as long as it issupplied as image data and may be exemplified by the following: imagesinput or acquired by an image input device, an image acquisition deviceand the like or images supplied from an image supply source (namely,images recorded on photographic films such as negatives and positives bythe conventional type of camera, as well as images such as photographicprints and printed images that are to be read by an image reading devicesuch as a scanner); images taken with a digital still camera or a videocamera; images taken with medical diagnostic image recording apparatussuch as an X-ray recording apparatus, a CR apparatus, a CT apparatus andan MRI apparatus; and images created by a computer and the like such asa computer graphics image (hereunder referred to as CG image). As willbe described later, the following images are also applicable: temporallycontinuous images successively taken with the conventional camera, thedigital still camera and the like (such images are hereunder referred toas dynamically changing images); dynamically changing images taken withthe video camera and the like (such images are hereunder referred to asvideo images); dynamically changing images broadcast on television(which are hereunder referred to as TV images); images having audioinformation as exemplified by such video images, TV images and CGimages; and images accompanied by message information as exemplified byletters, marks, facial images and character images.

[0086] The designated searching pattern 20 shown in FIG. 2 may becombinations of different specific figures or combinations of varyingsizes of the same specific figure. Aside from direct designation of thesearching pattern 20 by the user who either draws specific figures on animage display device with a mouse or from a keyboard or inputs concreteshapes using an image designating device such as a tablet, directdesignation may be performed as by designating through keywords such asletters, symbols and marks or by speech input (the input speech isrecognized and transformed automatically to concrete shapes that can bedesignated).

[0087] For example, instead of inputting specific figures in FIG. 2, theuser surrounds an approximate position on the tablet or the imagedisplay device and inputs letters, symbols or marks so that they fit inthe surrounded area. If the words “mountain” are input, they areautomatically transformed to TRIANGLE SEARCH MODE and processedaccordingly . Similarly, if the words “building” and “person” are input,they are transformed to RECTANGLE SEARCH MODE and FACE AND TRUNK SEARCHMODE, respectively. In the case of designation by symbols and marks, notonly service bureaus and other sites and centers that have a large storeof database for images accompanied by accessory information (and whichare commonly referred to as “database organizations”) but also usersthat utilize the database are preferably have the ability to set thesymbols and marks that can be input for indirect designation of thesearching pattern 20. In this way, the symbols and marks designated bythe user can be automatically transformed to specific figures orspecific principal subjects, persons and physical objects to bedescribed later on.

[0088] The specific figures to be used in the designated searchingpattern 20 may be any kind that corresponds to specific informationcontained in the image and, as set forth above, simple figures such ascircles, ellipses, triangles and rectangles are preferred since they areeasy to search for; however, these are not the sole examples that can beused in the invention and may be replaced by various other figuresincluding polygons having five or more sides, star shapes, star-shapedcharacters, irregular shapes, and straight or curved lines andcombinations thereof that can be found in the image. It should also benoted that the designated searching pattern 20 need not designate aspecific figure itself but may designate at least one attribute of thefigure selected from among its center position, size, direction,vertices and color.

[0089] The specific information to be used in the invention is notlimited to the specific figures described above and, as will be setforth later on, it may be replaced by a specific subject in the image,personal information consisting of the facial image of a specific personin the image and the name of that particular person which accompaniesthe image, audio information accompanying the image, message informationconsisting of text data as for letters and handwritten letters such assignatures or image data as for reference images such as marks, recordedimages and CG images, and editing data used in the creation of CG imagessuch as CG commands for drawing specific figures, specific colors andspecific physical objects.

[0090] On the pages that follow, the outline of steps for extractingpredetermined specific figures from the original image 10 shown in FIG.1 as a preliminary process for the image searching method according tothe example under consideration are described with reference to the flowdiagram shown in FIG. 4.

[0091] In the first step 401, the original image 10 is captured as imagedata from an image data supply source (not shown). In the next step 402,areas having similar shapes to predetermined specific geometric figures(specific figures for short) are extracted by the aforementioned patternmatching or edge tracing method. Subsequently, in step 403, theextracted areas are transformed to numerical data as related informationand their characteristic values to be described below are calculated. Instep 404, the calculated characteristic values are stored as accessoryinformation in memory means such as a database (not shown). In steps402-404, various characteristic values such as the center positions,sizes and directions of the specific figures in the original image 10are recognized and successively stored in the database as accessoryinformation.

[0092] The various characteristic values that can be used as accessoryinformation are exemplified by the following: if the specific figure isa circle, its center position and the diameter as its size are adopted;in the case of an ellipse, its center position and the major and minoraxes as its size are adopted; in the case of a triangle, the coordinatesof the three vertices are adopted; in the case of a rectangle, itscenter position and the widths as measured on the X- and Y-axes areadopted. Needless to say, the image searching method of the invention isin no way limited by the kind of characteristic values to be adopted. Ifthe original image 10 is a color image, related information such as thecolors of the specific figures may be used as accessory information.

[0093] The characteristic values of the specific figures such as theircenter positions, sizes and directions and related information such astheir colors are not the sole examples of the accessory information thatcan be used in the invention and may be replaced by the following whichwill be discussed later on: related information such as the position,size, direction and color of a specific subject or person; informationabout a specific figure, subject or person in a dynamically changingimage and the direction of movement of the specific figure, subject orperson between frames of the dynamically changing image; audioinformation per se; message information per se; and editing data per sefor creating CG images as exemplified by CG commands.

[0094] Table 1 shows an example of the result of extracting three kindsof specific figures from the original image 10 shown in FIG. 1. To bemore specific, Table 1 shows that the program for the image searchingmethod according to the example under consideration has extracted threeelliptical areas, one triangular area and two rectangular areas from theoriginal image 10. As already mentioned, one of the elliptical areas andone of the rectangular areas have been erroneously detected. TABLE 1 No.Ellipse (A) Triangle (B) Ellipse (A) coordinate Rectangle (C)coordinates coordinates of the coordinates No of the direction of of thethree of the width on Width on i center size major axis center pointscenter X-axis Y-axis 1 (a11, a21) a31 a41 (b11, b21) (b31˜b81) (c11,c21) c31 c41 2 (a12, a22) a32 a42 — — (c12, C22) c22 c42 3 (a13, a23)a33 a43 — — — — — 4 — — — — — — — —

[0095] The flow of steps in the process of searching through images onthe basis of the stored data in accordance with the image searchingmethod of the invention is now described with reference to FIG. 5.

[0096] First, in step 501, the designated searching pattern 20 is inputas a searching condition. In the next step 502, the input pattern 20 istransformed to numerical data (characteristic values) and tabulated inthe same manner as typically shown in Table 1. In this step,characteristic values such as the position, size and direction of eachspecific figure and its color may be directly input. In the next step503, the original image to be searched through is read from its database(DB). Then, in step 504, the input original image is compared forverification with the designated searching pattern 20 tabulated in step504.

[0097] The verification is performed through the successive rows andcolumns in the designated searching pattern table. If a plurality ofcharacteristic values in the designated searching pattern are arrangedin a single row, the characteristic values in the first designatedpattern that are arranged in the respective columns of the first row arecompared with the characteristic values appendant as accessoryinformation to the original image read from the database and a check foragreement is made in step 505. If the answer is yes, the characteristicvalue is retrieved in step 506 as the result of search.

[0098] If no agreement is found in step 505, step 506 is bypassed andthe process goes to step 507 (MOVE to the next row). In the next step508, the designated searching pattern table is checked for the presenceof the next row; if the answer is yes, the process returns to step 504and a check for verification is made with respect to the next row byrepeating steps 504-508. If no next row is found in step 508,verification of one original image to be searched through ends and theprocess goes to step 509 (MOVE to the next original image to be searchedthrough). In the next step 510, a check is made to see if the nextoriginal image exists. If the answer is yes, the process returns to step503 and the next original image is searched through. If no next originalimage is found in step 508, the search through all original images ofinterest ends.

[0099] This is how the searching method of the invention is implemented.

[0100] If a plurality of characteristic values in the designatedsearching pattern are arranged in a column in the table, theabove-described verification process is performed through successivecolumns; if the characteristic values are arranged in blocks of certainrows and columns, the verification is performed block by block.

[0101] A word must be said on the sentence reading “If the answer is yesin step 505, the characteristic value is retrieved in step 506 as theresult of search”. The actual process is not this simple becausecomplete agreement seldom occurs. Preferably, the degree of agreement tobe defined below is calculated in step 504 and only in the case wherethe calculated degree of agreement exceeds a predetermined threshold,the characteristic value is found to “agree” with the accessoryinformation.

[0102] It should be noted here that the designated searching pattern 20that has been input and designated by the user is transformed totabulated data, as typically shown in Table 2, and stored in a memoryunit (not shown). Table 2 shows only data for an ellipse but similardata are constructed for other figures. On the basis of the thusconstructed data, the above-described process is performed to search foran original image is search for that fits the designated searchingpattern. TABLE 2 No Ellipse coordinates of direction of the j the centersize major axis 1 (ta11, ta21) ta31 ta41 2 (ta12, ta22) ta32 ta42

[0103] Given below is an example of the mathematical formula forcalculating the degree of agreement in the case of an ellipse:$\begin{matrix}{{D\left( {I,J} \right)} = {F\left\{ {{{k1} \times \left( {\left( {{a1I} - {ta1J}} \right)^{2} + \left( {{a2I} - {ta2J}} \right)^{2}} \right)} + \quad {{k2} \times \left( {{a3I} - {ta3J}} \right)} + \quad {{k3} \times \left( {{a4I} - {ta4J}} \right)}} \right\}}} & \left( {{Eq}.\quad 1} \right)\end{matrix}$

[0104] where the value of F{X} represents a characteristic thatincreases with decreasing input value X; I represents the number ofcharacteristic values contained in accessory information and ranges from1 to Imax; J represents the number of searching patterns and ranges from1 to Jmax; k₁- k₃ are coefficients.

[0105] The score point for J-numbered searching pattern is expressed by:

P(J)=max(D(I,J))

[0106] The score point for the entire screen is expressed by:

Ptotal (A:ellipse)=Σ(P (I))

J=1−Jmax.

[0107] For other figures than ellipse, the same procedure is taken tocalculate the degree of agreement in step 504.

[0108] Consider the case where the degree of agreement is Ptotal (A),Potal (B) and Ptotal (C) for ellipse, triangle and rectangle,respectively; the degree of agreement in the frame shown in FIG. 2, orPtotal (ALL), can be calculated by summing up Ptotal (A), Ptotal (B) andPotal (C).

[0109] It should be remembered that the specific figures to be used inthe invention are by no means limited to the geometric figures describedabove and many diverse shapes including other polygons, characters suchas stars, straight lines, curved lines and combinations thereof may besubstituted

[0110] desired, the results of calculation may be used to display thesearched images in a thumbnail layout in the increasing or decreasingorder of the degree of agreement.

[0111] Two different modes, AND conditional and OR conditional, may beset for the designated searching pattern, as illustrated below.

[0112] (1) AND conditional mode: Function F{X} is designed to have suchcharacteristics that it takes a negative value when input X is no morethan a specified value. For example, F{X} takes a positive value ifthere is a designated figure in the designated searching pattern buttakes a negative value if there is no such figure. If there is no hit,the overall score point for the degree of agreement decreases.

[0113] (2) OR conditional mode: Function F{X} is designed to have suchcharacteristics that it takes either zero or a positive value. Forexample, F{X} takes a positive value if there is a designated figure inthe designated searching pattern but takes zero if there is no suchfigure. Even if there is no hit, the overall score point for the degreeof agreement does not decrease.

[0114] In the foregoing example, characteristic values such as position(e.g. center position), size and direction in all designated searchingpattern 20 are used as searching conditions to make comparison andverification with the accessory information accompanying the inputoriginal image. The present invention is by no means limited to thisparticular case and only the presence or absence of designated searchingpatterns may be used as the searching condition or, alternatively, atleast one element or item selected from among such characteristic valuesas position, size, direction and color may be used as the searchingcondition. If desired, the characteristic values may be used as thesearching conditions in combination with the colors (shades) of thedesignated patterns. In a special case, at least one designatedsearching pattern may be used as the searching condition.

[0115] If the presence or absence of designated searching patterns is tobe used as the searching condition or if only a small number ofelements, items or searching patterns are designated, less time is takento complete the searching process and more original images are hit andretrieved, thus increasing the likelihood that the desired originalimage can be found but, on the other hand, the precision in searchingdecreases. If more elements, items or searching patterns are designated,the precision in searching improves to increase the chance of hittingand retrieving only the desired original images; on the other hand, ittakes a longer time to complete the searching process and if thedesignated elements, items or searching patterns are not correct enough,the user may end up with hitting no original image. Under thesecircumstances, the number of elements, items or searching patterns to bedesignated need be selected as appropriate considering the precision insearching (i.e., the chance of success in hitting the right originalimage).

[0116] In a preferred case of user interface for designation andsearching purposes, the user may have an option to designate a suitablemode from the following three: 1) a check is made for only the presenceor absence of specific information such as specific figures; 2) inaddition to this check, characteristic values of specific figures suchas their positions, sizes and directions, as well as their colors arealso used; and 3) as a means of associating various items and elementssuch as the characteristic values and colors of specific figures, theyare used either independently or in combination. To give an example,coefficients k₁ and k₂ in Eq. 1 may be adjusted to zero and excludedfrom the calculation of the degree of agreement.

[0117] The image searching method according to the example justdescribed above offers the advantage that an image complying with thegiven instruction can be searched for by a simple way of givinginstructions such as drawing a rough sketch of figures on a tablet,inputting the names or symbols of figures from a keyboard or inputtingthese instructions by speech. In the present invention, figures or theirnames (in language) that have been entered by simple ways of givinginstructions or designation by symbol or speech are transformedautomatically to specific figures, further transformed to numerical dataconsisting of the required characteristic values and changed in colorshades and the like.

[0118] We next describe another example of the first embodiment of theinvention. In the first example, specific figures are used as specificinformation and the positions, sizes, directions, colors, etc. of thespecific figures to be searched through are used as accessoryinformation. In the second example, a specific principal subject(hereunder referred to as a specific subject and a human individual isgiven as an example in the following description) is used as specificinformation and its position, size, direction, color, etc. are used asaccessory information in the searching process.

[0119]FIG. 6 shows a specific subject 60 that is designated to start thesearching process. In the example under consideration, the subject isset as a person and its presence or absence or at least one elementselected from among its position, size, direction and color is specifiedusing a simple figure, say, an ellipse 61.

[0120]FIG. 7 shows the result of searching that corresponds to thedesignation of the person. In the case under consideration, searching isdone on the basis of only the position and the size of the person to hitthe desired image 70.

[0121] The image searching method of the second example offers theadvantage that if a person is to be searched for, an image that complieswith the given instruction can be searched for by a simple way of givinginstructions such as drawing on the tablet a rough sketch of the personper se or at least one element selected from among his or her position,size and direction or inputting from the keyboard the presence orabsence of the person or at least one element selected from among his orher position, size and direction.

[0122] Besides the position and size of the person, informationexpressing his or her clothes by color may be added; if the color shadeof the trunk of the person is checked in the searching process, theefficiency of search may potentially improve by large extent. Ifdesired, the characteristics of more than one person may be combined.

[0123] In certain fields of application, specific figures or subjectsmay be substantially fixed in position. In that case, their sizes eitheralone or in combination with their directions, optionally together withtheir colors, may be recorded as accessory information.

[0124] If specific figures or subjects are substantially fixed in size,their positions either alone or in combination with their directions,optionally together with their colors, may be recorded as accessoryinformation.

[0125] We next describe yet another example of the present invention. Inthe previous examples, static images are searched through and theinitial positions of specific figures or specific subjects (persons) andthe directions of their movements are used as accessory information anddesignated as a key in the searching process. In the third example, aperson 81 is set as the specific subject and the direction of itsmovement (vector) 82 is designated by an arrow.

[0126]FIG. 8 shows a specific subject designated to search throughdynamically changing images.

[0127]FIG. 9 shows the result of searching that corresponds to thedesignation of the person. In the case under consideration, searching isdone on the basis of only the initial position and size of the person tohit the desired dynamically changing images 90 a and 90 b have beensearched for.

[0128] To register accessory information in the third example, adesignated figure (specific figure) or a designated subject (specificsubject) that have been extracted within a frame of dynamically changingimage is traced for the movement it has made through a specified numberof frames and the obtained vector information is stored as part of theaccessory information. For searching purposes, matching with theaccessory information is performed on the basis of the designated figureand the vector for its movement (representing either its size ordirection or both).

[0129] In another method, accessory information can be constructed frameby frame and search is made in the following manner. If a designatedfigure (specific figure) or a designated subject has been detected frameby frame, matching with the accessory information is done to make acheck of whether the same designated figure is found in thecorresponding position (which can be determined from the designatedmotion vector) in a frame after the lapse of a specified time from thecurrent frame (or in an intermediate frame).

[0130] In the image searching method of the third example, the programdecodes the designated information for the input scene into simpleshapes and their movements and the degree of agreement with theaccessory information for each of the dynamically changing images in thedatabase is calculated. The results of calculation are output (in athumbnail layout) by indicating the frames of relevant dynamicallychanging images in the order of score points.

[0131] The image searching method of the third example offers theadvantage that if a person in a dynamically changing image is to besearched for, a dynamically changing image that complies with the giveninstruction can be searched for by a simple way of giving instructionssuch as roughly showing its initial position and the direction of itsmovement on the tablet or inputting them from the keyboard or by speech.As a result, when editing videos, images can be easily searched throughby rough scene description.

[0132] A further example of the invention is described below. In theprevious examples, images that do not contain audio information aresearched through but in the fourth example, images such as dynamicallychanging images that contain audio information are searched through andoutlines of the audio information are contained as part of the accessoryinformation; for searching purposes, such outlines of the audioinformation are designated.

[0133] Take, for example, a dynamically changing image that contains ahuman individual as a subject. Audio information is picked up inaddition to motion information and its contents are stored in thedatabase by speech conversion. For searching purposes, a set of keywordsconsisting of SUBJECT: PERSON and AUDIO INFORMATION: SCHOOL may bedesignated to search through scenes in which a person is found and whichare related to the word “school”. The images to be searched through neednot be dynamically changing images but static images accompanied byaudio information may be substituted.

[0134] The image searching method of the fourth example offers theadvantage that if the images to be searched through have audioinformation, it may be used to search for an image that complies withthe given instruction. Since search can be made not only from themovement of the subject but also from the contents of his or her speech,there are great benefits to the case of editing videos. Needless to say,audio information is not the only information that can be used in thefourth example and various kinds of information about specific figuresor a specific principal subject (person) may be additionally designated.

[0135] We describe a still further example of the invention. In thisfifth example, a specific person in the image is identified by a personidentifying treatment which comprises the steps of preliminarilyregistering the facial images of specific persons and their names asaccessory information and searching for the desired image by designatingone of the registered persons with reference to his or her facial imageand name.

[0136] Specifically, the customer preliminarily registers informationabout N persons in the form of the combination of name and facial image.To identify a specific person in the image by a person identifyingtreatment, there can be applied known techniques such as the onesdescribed in Kosugi et al., “Searching Through Similar Facial ImagesUsing Mosaic Features of Varying Densities”, Journal of the Institute ofElectronics, Information and Communication Engineers, vol. J80-D-11 No.8, pp. 2218-2224 and Soma et al., “Verification of Facial ImagesAcquired by Extremely Different Processes”, Journal of the Institute ofElectronics, Information and Communication Engineers, vol. J80-D-11 No.8, pp. 2225-2231.

[0137] Images submitted to a service bureau may be subjected to theabove treatment to extract the face and identify the person of interestso as to construct accessory information. To be specific, accessoryinformation telling who are in each frame is added to the image data andstored. Needless to say, information about the position and size of theperson may be included as part of the accessory information. In the casewhere the customer asking for image search designated the name of aperson, an image frame showing the right person is retrieved on thebasis of the accessory information.

[0138] In the image searching method of the fifth example, the facialimage of the person to be searched for and his or her name are keyed toeach other when they are transformed to accessory information and animage that complies with the given instruction can be searched for bymaking use of that information. For example, a photo full of memory canbe easily searched for by using the name of a family member as akeyword. Needless to say, not only static images but also dynamicallychanging images can be searched through by the method of the fifthexample.

[0139] When searching through images having audio information, variouskinds of audio information may be combined with each other.

[0140] If the service bureau is connected to the personal computer of acustomer by communication lines, the above-described treatment can beperformed on-line at a faster speed.

[0141] The following is a yet another example of the invention. In theprevious examples, it is chiefly the recorded images that are to besearched through. In the sixth example, computer graphics (CG) imagestypically created with a personal computer are searched through and ascene is designated by simple figures, approximate colors or a roughsketch of specific figures and search is made using CG image creatingand editing data.

[0142] In the sixth example, editing data relevant to the creation of CGimages is used as specific information; at the same time, it is used asaccessory information to the CG images and stored in the database. Inthe database organization, the CG images and the editing data are storedas a set. The editing data are preferably CG commands that represent thesimple figures, approximate colors and specific objects such as humans,animals, automobiles and buildings in the CG images and which wereentered to create those figures, colors and objects.

[0143] When searching through CG images in the example underconsideration, the user (customer) designates a scene by simple figures,approximate colors or specific objects such as humans, animals,automobiles and buildings. The designation is preferably done eithermanually or by means of drawing software, CG software and the like.

[0144] In the database organization, a database processor is used totransform the scene designating data from the user into CG commands. Inthis step, the shapes of the figures the user adopted to designate thescene, characteristic values such as position, size and direction,colors, etc. are preferably classified into predetermined patterns.

[0145] In the next step, the degree of agreement or similarity betweenthe transformed CG commands and the CG editing data in the database thataccompanies the CG images as accessory information is calculated andmatching with the accessory information is performed. The results ofcalculation may be output (in a thumbnail layout) by indicating theframes of relevant CG images in the order of score points.

[0146] The image searching method of the sixth example offers theadvantage that if the data for the creation and editing of the CG imagesto be searched through is used as accessory information, the volume ofcalculations that must be made in the process of searching through CGimages can be sufficiently reduced to ensure rapid and precise searchingfor a CG image that complies with the user instruction. Needless to say,the CG images to be searched through may be static or dynamicallychanging. If dynamically changing CG images are to be searched through,specific figures or objects and their motion vectors (representingeither their size or direction or both) are preferably designated asdescribed above. When searching through images having audio information,various kinds of audio information may be combined with each other.

[0147] We next describe a still further example of the invention. Inthis seventh example, images having message information are searchedthrough and the message information that accompanies the images forsearching purposes are checked to perform the intended search. As in thealready-described case of searching through images having audioinformation, the message information accompanying the image is used asspecific information and, at the same time, it is used as accessoryinformation to the images and stored in the database.

[0148] The accessory information to the images is preferably the textdata synthesized with the images as message information or image datasuch as marks, recorded images and CG images. As in the above-describedcase of audio information, text data that accompany the images ratherthan being contained in them or image data, for example, text datarecorded (synthesized) on the back side of printed images or in thewhite margins of printed images if they have such white margins or imagedata may be substituted.

[0149] Exemplary text data are printed character data as output from aword processor and handwritten character data such as signatures; ifdesired, character data obtained by extracting letters from images maybe substituted.

[0150] To search through images in the example under consideration, theuser (customer) designates the message information by letters per se,their contents, the shape of handwriting (as in the case of signatures)or selects one of the reference images registered as messageinformation; after the designation, matching is performed.

[0151] Consider, for example, the mode of searching through letters perse; as in the already-described case of designating the characteristicvalues of specific shapes, their colors, etc. to make search using suchspecific figures, the presence or absence of letters or at least oneelement selected from among the positions of the letters, their sizes,directions, colors, etc. is designated and matching is performed bycalculating the degree of agreement and selecting the image thatsatisfies the requirement for agreement. If the contents of letters areto be designated in the LETTER SEARCH mode, the contents of letters suchas designated kanji (Chinese characters) are interpreted and matchingfollows. In the case of handwritten letters such as signatures, theshape of the handwriting may be designated for matching purposes.

[0152] If search is to be performed using reference images, the userdesignates the area of a reference image from a plurality of referenceimages such as marks, recorded images and CG images that have beenregistered as message information, or selects one of the preparedpictorial patterns or otherwise selects one or more reference images;after this designation, image matching is performed.

[0153] If desired, the results of calculation of the degree of agreementin the matching process may be output (in a thumbnail layout) byindicating the frames of relevant images in the order of score points.

[0154] The image searching method of the seventh example offers theadvantage that if text data such as letters or reference image data areused as message information that accompanies the images to be searchedthrough, an image that complies with the given instruction can besearched for by a simple way of designation such as designating thecharacteristic values of letters per se, their colors or contents andthe shape of handwriting or by another simple way of designation such asdesignating a reference image area from a plurality of preliminarilyregistered reference images or selecting one of the prepared pictorialpatterns.

[0155] If the user wants to display an image as the result of searching,the area that meets the designated searching condition(s), exceptingmotionless parts such as the background, may be displayed in preferenceto other areas. In another effective way, the area complying with thedesignated searching condition(s) may be displayed as enhanced over theother areas after it has been subjected to special image processing suchas luminance enhancement or blinking.

[0156] By adopting this display method, the data transfer volumerequired to transfer the hit images to a monitor or the like can besignificantly reduced, so the response speed of display is sufficientlyincreased to provide higher operating efficiency. In addition, thedesired image in the display screen becomes easier to look at, againproviding higher operating efficiency.

[0157] When the desired image is to be displayed, it is preferred tocheck the performance of the display device and let it display the imageafter the image data has been processed to meet the specific performanceof the display device. According to another effective approach, theareas that have been searched through are managed on the database areaby area and the priority of reference in search mode or the imageresolution for data storage is set in accordance with access frequency.

[0158] This method may be implemented as follows: when the customeraccesses the database center (organization) from his or her mobileterminal, information about the display performance of the mobileterminal is transferred with the ID and the data being transferred fromthe center is subjected to various image data processing steps such asimage compression, binarization, conversion to line art and trimming(cutting only the principal subject).

[0159] By thusly performing efficient processing for image display, thedata traffic volume can be reduced, the time to image display at themobile terminal is shortened and the displayed image becomes easier tolook at.

[0160] To manage the image data furnished with various kinds ofaccessory information, it is preferred to divide the image into aplurality of areas for the respective subjects and register theindividual areas with labels attached. If only the labeled areas aresliced and stored in file format, there are great benefits to reductionof data volume and faster search speed. Another applicable method is bystoring only the images in the labeled areas at high resolution whilestoring the other areas after image compression.

[0161] Various kinds of accessory information may accompany the areas ofthe image and they include the name of image file, its position (centerand contour pixels), attributes such as color, and cumulative number ofaccesses/number of hits. If the memory hierarchy in the storage means iscontrolled in accordance with access frequency using such accessoryinformation, the search speed can be further increased. Data having highfrequency of use becomes much easier to use if it is saved individually.

[0162] Having described the image searching method according to thefirst embodiment of the invention, we now explain the image processingmethod according to the third embodiment of the invention which appliessaid image searching method. According to the first approach, at thetime of first printing (so-called “simultaneous printing”) or a previousreprinting at the service bureau (the term “previous printing” ishereunder used to represent the two printing events), specificinformation such as specific figures or a subject in the printed imageis extracted and the extracted specific information and relatedinformation (characteristic values and colors) are used as accessoryinformation and stored in memory means such as the database togetherwith the image data for printing and the parameters for imageprocessing.

[0163] If it receives a request for reprinting a film containing theimage or making a later reprint of the same (the term “later printing”is used to represent the two printing events), the service bureauextracts specific information such as specific figures or a subject fromthe image frame designated by the customer and performs later printingunder the image processing conditions (processing parameters) employedto perform the previous printing of the image having a high degree ofagreement (in excess of a predetermined threshold) with the previouslystored accessory information.

[0164] According to this image processing method, the image processingconditions adopted in the previous printing can be searched from theimage submitted to the service bureau at the time of later printing andprints are produced that have high enough fidelity to satisfy thecustomer. The above-described processing is not only applicable tophotographic films; it is equally applicable to smart media such as DSC(digital still camera) or other captured image recording media havingprinting image data recorded thereon, and image recording media such asMOs and photo CDs.

[0165] Now consider the case where the customer placing an order forreprinting has submitted to the service bureau not the photographic filmor the captured image data on captured image recording medium but thephotographic print made at the time of simultaneous printing. Theservice bureau reads the print with a scanner, executes the extractionof specific information such as specific figures for image searching,uses the extracted specific information and related information as theconditions for searching through the image data in the database thathave been read from the photographic film recorded at the time ofsimultaneous printing, calculates the degree of similarity (performsmatching), picks up the relevant frame, and performs printing. In theprinting step, the image data corresponding to the relevant frame andwhich was input at the time of simultaneous printing and the associatedprocessing parameters may be retrieved for printing operations.

[0166] The second approach is applied to ensure that there will be novariations in printing a single film strip (so-called photographic filmsof a single case), or a single captured image recording medium, or oneor more photographic films or captured image recording media for whichan order was placed by the user (customer) at a time (so-called singleorder). The rationale of this approach derives from the fact that ifphotographic films or captured image recording media of a single case ora single order (the term “a single order” is used to represent the twokinds) have compositions characterized by similarity in specificinformation such as specific figures or subjects (the term “composition”means the layout of different figures, their sizes, directions andcolors), image processing is preferably adjusted to ensure thatindividual frames within a single order have uniformity in the finishingcolor and density.

[0167] According to the second approach, similar images aredifferentiated on the basis of their compositions, so its performance isless sensitive to the effects of color and density; as a result, thereis no possibility that prints having extremely different colors anddensities are produced from photographic films or captured imagerecording media of a single order and no customer will leave the servicebureau dissatisfied.

[0168] Described above are the fundamental features of the imagesearching method according to the first embodiment of the invention andthe image processing method according to the third embodiment.

[0169] We now describe in detail the image searching method according tothe second embodiment of the invention by referring to FIGS. 10-15B.

[0170]FIGS. 10 and 11 are diagrams for illustrating an image searchingmethod according to an example of the second embodiment of theinvention. To be specific, FIG. 10 shows an original image 110 on aphotographic film that was taken with a GPS incorporating camera. Theoriginal image 110 is captured with an image reading device such as ascanner and its image data is stored in memory means such as thedatabase. FIG. 11 shows a designated searching pattern 120 which theuser in an attempt to search through the original image 110 has inputwith a mouse or the like so that it is displayed on the screen of aterminal device or an image display device.

[0171] Referring to the original image 110 shown in FIG. 10, numeral 111indicates a human being as a principal subject, 112 indicates a mountain(hereunder called “Mt. X” for convenience sake) which is a subjecteffective to specify the shooting place (the place where the originalimage was taken) or the shooting position (the position in which theoriginal image was taken), and 113 indicates a famous building(hereunder called “Z Bldg.”) which is also a subject effective tospecify the shooting place or position.

[0172] The designated searching pattern 120 shown in FIG. 11 was inputby the user to express his or her desire of “searching for a sceneshowing a mountain called so-and-so around area T1 and also showing abuilding called so-and-so around area T2”. Numeral 122 is an indicationof the position where the mountain is and 123 is an Indication of theposition where the building is.

[0173] As preliminary steps for the image searching method according tothe example under consideration, the position and name of a subjectwhose position in the original image 110 shown in FIG. 10 can bespecified are specified and the outline of an example of these steps isdescribed below with reference to the flow diagram in FIG. 12.

[0174] As FIG. 12 shows, when the image is captured, typically with ascanner, in step 101, the accompanying data, i.e., the aforementionedGPS information about the shooting time, as well as the camera directionand magnification rate relevant to the shooting time are also captured.In the next step 102, the captured GPS information is superposed onalready available map information, whereupon the shooting location onthe map is specified.

[0175] In the nest step 103, the camera direction and magnification raterelevant to the shooting time are superposed on the map information,whereupon the subject in the image is specified so that its position,name (e.g. place name), size on the screen and like items are specified.In this process, a simulated image may be created on the basis of cameradirection and magnification rate in the relevant position on the map andsubjected to pattern matching with the recorded image so as to specifythe subject. In the last step 104, the positions, names (e.g. placename), sizes, etc. of the specified subjects are keyed to the image dataand stored successively in the database.

[0176] Table 3 shows an example of accessory information that has beencreated on the basis of shooting position information added to theoriginal image and which is to be further added to the image data forthe original image.

[0177] As shown, the subjects such as Mt. X and Z Bldg. whose positionsare already known from the GPS information about the shooting time andcamera direction and magnification rate relevant to the shooting timeare listed together with their positions on the screen (or addresses onthe image data stored in the database), shapes, sizes, positions on themap and names (e.g. place name).

[0178] In the illustrated case, the subjects in the original image arespecified and their positions, names (e.g. place name) and sizes areused as added information. This is not the sole case of the inventionand the place where the original image was taken (the place representedby the original image) may be specified whereas the position, name (e.g.place name), etc. of the specified shooting place are used as accessoryinformation. Alternatively, information about the specified subjects andshooting place may both be used as added information. In this case, theshooting place may be specified as a single point on the map or it maybe roughly specified as a local area.

[0179] The shooting date, information about the shooting position(latitude, longitude and place name) and other camera information mayalso be registered in the database as accessory information to therecorded original image. TABLE 3 Screen information Map coordinatesinformation Subject No of the vertical latitude, name i center sizeposition longitude — 1 (a11, a21) A31 A41 (b11, b21) Mt. X 2 (a12, a22)A32 A42 . . . Lake Y 3 (a13, a23) A33 A43 . . . Z Bldg. 4 (a14, a24) A34A44 . . .

[0180] In the example under consideration, the step of listing theabove-mentioned accessory information is performed automatically, so theuser can save the trouble of inputting the names of subjects assearching data when registering the image data in the database.

[0181] The user starts the image searching process by inputting variousitems as searching information, or searching reference data, and theyinclude the position and name of the place where the original image wastaken or the position information (latitude and longitude) about thesubjects in the original image on the screen and their names (e.g. placename). Such reference data may be input by manipulation of a mouse orthe like on the display screen of a terminal device as in the case ofdesignating the searching pattern 120 shown in FIG. 11. Indirectposition designation can be made by specifying the names of mountain andbuilding.

[0182] Image searching is then performed on the basis of the thuslystored data. An example of the image searching process in the case underdiscussion is described below with reference to FIG. 13.

[0183] The process starts with step 201, where a designated searchingpattern of the type shown in FIG. 11 is captured as a searchingcondition. In the next step 202, the captured pattern is superposed onmap information so that it is transformed to user designated searchinginformation, or position information about the subjects of interest. Inthis instance, the position information for each subject as transformedfrom the designated searching pattern is preferably tabulated. Thesesteps may be skipped if the user has directly input at least one itemselected from among the position information, name (e.g. place name) andsize of the subject or shooting place.

[0184] In the next step 203, the original image to be searched throughis captured from the database (DB). Subsequently, in step 204, thecaptured image is compared for verification with the positioninformation in a table format of the designated searching pattern 20. Asin the case of comparison for verification with the designated searchingtable used in the already described first embodiment of the invention,verification is performed through the successive rows and columns in theposition information table. For instance, in step 204, the positioninformation for each subject given in a row or column in the table iscompared for verification with the position information appending asaccessory information to the original image read from the database and acheck for agreement is made in step 205. If the answer is yes, theposition information is retrieved in step 206 as the result of search.

[0185] If no agreement is found in step 205, step 206 is bypassed andthe process goes to step 207 (MOVE to the next row). In the next step208, the position information table is checked for the presence of thenext row or column; if the answer is yes, the process returns to step204 and a check for verification is made with respect to the next row orcolumn by repeating steps 204-208. If no next row or column is found instep 208, verification of one original image to be searched through endsand the process goes to step 209 (MOVE to the next original image to besearched through). In the next step 210, a check is made to see if thenext original image exists. If the answer is yes, the process returns tostep 203 and the next original image is searched through. If no nextoriginal image is found in step 208, the search through all originalimages of interest ends.

[0186] This is how the searching method of the second embodiment of theinvention is implemented.

[0187] A word must be said on the sentence reading “If the answer is yesin step 205, the characteristic value is retrieved in step 206 as theresult of search ”. The actual process is not this simple becausecomplete agreement seldom occurs. Preferably, the degree of agreement tobe defined below is calculated and only in the case where the calculateddegree of agreement exceeds a predetermined threshold, thecharacteristic value is found to “agree” with the accessory information.

[0188] An example of the method of calculating the degree of agreementis by using the “coordinates of the center”, “size” and the “verticaldirection” listed as accessory information in Table 3, calculating thedegree of agreement (or similarity) for a plurality of candidates forsubjects in the database of the original images and outputting thesearched images in the decreasing order of score points for the degreeof agreement.

[0189] According to the image searching method of the example underconsideration, an image that complies with the given instruction can besearched for on the basis of the designated searching pattern 120 theuser has input, typically with a mouse, on the screen of a terminaldevice or a like image display device or on the basis of the positioninformation (latitude and longitude) about the subjects in the screen,their names (e.g. place name) and the like that have been input assearching information from a keyboard.

[0190] We next describe another example of the second embodiment of theinvention. In the previous example, the position and name (e.g. placename) of the shooting position or subject that have been created fromthe original image and the associated shooting information (positioninformation) and map information are used as accessory information. Inthe following description of the example under consideration, scenes aredifferentiated or verified using the image's depth information (whichrepresents the depth in the image or the distance from the camera) asaccessory information.

[0191] In this example, depth information for various subjects in theimage can be derived using shooting position information and mapinformation but it should be understood that the example can also beapplied to the case where no such shooting position information or mapinformation is used. In this alternative case, the depth information forvarious subjects in the image may be derived from multi-stage focusedimages or stereoscopic images that are within well-known cameratechnology.

[0192]FIG. 14 shows an example of designated searching pattern 130 whichcontains depth information. In the case shown in FIG. 14, a circle 131corresponds to a person, an area far back above corresponds to amountain (and indicated by a horizontally elongated ellipse 132) and arectangle 133 on the right corresponds to a building.

[0193] Stated more specifically, FIG. 14 shows a scene in which theperson 131 stands at a distance of about 5-10 m, the building 133 islocated on the right side far about 100 mm back, and the mountain 132 isfarther backward of the person by a distance of about 1000 m. Ifdesired, the designation of the positions of the subjects in the screenshown in FIG. 11 may be combined with the layout shown in FIG. 14 whichconsists of the shapes of the subjects.

[0194] By means of the layout shown in FIG. 14, the image to be searchedfor is designated by the user who expresses the depth of the scene foreach of the roughly divided areas. Calculation is then made for thedegree of agreement between the contents of the designated image and aplurality of candidates for the subjects in the database of the originalimage and the searched images are output in the decreasing order ofscore points. If desired, subjects shown as patterns may be arranged ina virtual map space of the type constructed by combining FIGS. 11 and 14so that an outline of the depth or map is designated.

[0195] According to the image searching method of the example justdescribed above, the user adds a 2D layout of subjects to the inputsearching pattern 120 and this offers the advantage of enabling mucheasier searching for an image that complies with the given instruction.

[0196] The image searching method of this example is also applicable todynamically changing images. In this case, information about the motionof dynamically changing images between frames can be utilized.

[0197] We next describe a further example of the second embodiment ofthe invention. The following description of this third example concernsthe case which employs a layout showing the relative positions ofsubjects. The layout may be the one shown in FIG. 14 but any otherlayout can be used as long as it shows the horizontal positions ofsubjects such as a mountain and a building relative to a person which isthe principal subject.

[0198] Position information that can be used here may be theaforementioned GPS information or the result of position determinationby PHS (standing for the personal handyphone system which, in the stateof the art, is much inferior to GPS in terms of precision) or more roughposition designating information (such as the indication of “xx-chome,yy-machi, zz-shi).

[0199] Thereafter, matching is performed by verifying the layout on themap data against the layout used in designation and the degree ofagreement is calculated for each candidate of subjects. If there is noinformation about camera direction, it can be simulated by rotating thecamera direction through an angle of 360 degrees at predeterminedintervals. If positional precision is low, simulation can be made byparallel shifts within a predetermined range.

[0200] According to the example under consideration, simple instructionssuffice to search for images using topography and other information,thus providing efficiency for the search.

[0201] When the above-described image search is performed by the priorart technology, position information and the direction of viewpoint areoften provided in the manner shown in FIG. 15B, where the positioninformation is the camera position (GPS information) and the directionof viewpoint is the shooting direction (camera information). If a formatlike the one shown in FIG. 15A is obtained, namely, if information isobtained about the position of a subject and the shooting direction,this can be used to perform the following operation.

[0202] Since the user already knows what was taken by the camera, therecorded image of the subject is retrieved from the available imagedatabase and a computer graphics (CG) image is created on the basis ofthe retrieved image; the CG image is compared with the actually recordedimage and after the necessary adjustments such as position adjustmentand size adjustment are made, the difference between the two images istaken and stored as data, optionally after compression.

[0203] The CG image is not the sole example of the simulated image andother images may be employed, such as a real image that was takenpreviously and which is believed to be the best approximation in thelight of position information and camera shooting information, and thecombination of such real image with the CG image. Whichever kind ofimage is used, the volume of data to be stored can be reduced.

[0204] This is the basic features of the image searching methodaccording to the second embodiment of the invention.

[0205] The image searching methods according to the first and secondembodiments of the invention and the image processing method accordingto the third embodiment of the invention can be implemented by means ofan image searching apparatus 302 and a print system 300 using thisapparatus 302, respectively. These are shown in FIG. 16.

[0206] The print system 300 shown in FIG. 16 comprises the imagesearching apparatus (which is hereunder simply referred to as searchingapparatus) 302 for implementing the image searching method of theinvention as mentioned above; a photoprinter 304 that is connected tothe searching apparatus 302 and that implements the image processingmethod of the invention in corporation with the searching apparatus 302;and a user terminal 306 that is connected on-line to the searchingapparatus 302 by a communication network such as Internet and that canimplement the image searching method and image processing method of theinvention and also make a request for print output.

[0207] First of all, the image searching apparatus 302 comprises animage database (which is hereunder simply referred to as database or DB)308; an accessory information acquiring section (which is hereundersimply referred to as acquiring section) 310; an accessory informationdesignating section (which is hereunder simply referred to asdesignating section) 312; an image searching section (which is hereundersimply referred to as searching section) 314; and a display section 316.The data base 308 stores not only the image data of an image ofinterest, but also the specific information per se or the relatedinformation that is found in or accompanies the image as the accessoryinformation of the image data. The acquiring section 310 acquires thespecific information per se or the related information as the accessorydata for storing in the database 308. The designating section 312designates at least a portion of the accessory information as thesearching condition. The searching section 314 searches the database 308based on the searching condition designated by the designating section312 and determines the degree of agreement of the accessory conditionthat is appendant to the selected image data with at least a portion ofthe accessory information used as the searching condition therebyreading an image having a degree of agreement above a predeterminedvalue. The display section 316 displays the result of the search made bythe searching section 314 and the read image.

[0208] The photoprinter 304 comprises an image input section 318, animage processing section 320 and a print output section 322. The imageinput section 318 reads the image data of an image recorded on a film Fsuch as a negative or reversal film or the image data of an image takenwith a digital camera and stored in an image data recording medium R, orreceives the image data from an image data supply source such as a CGimage creating device 317. The image processing section 320 analyses theimage data received from the image input section 318 to set the imageprocessing conditions for use in processing image data with a view tocreating a print of high quality and subjects the image data receivedfrom the image input section 318 or the searching apparatus 302 to imageprocessing according to the image processing conditions previously setor received from the searching apparatus 302. The print output section322 exposes a photosensitive material or another recording materialusing the image data processed in the image processing section 320 torecord an image (or latent image), which is then developed to be outputas a print P. In the illustrated case, the image processing section 320transfers the image data to the acquiring section 310 and receives theaccessory information obtained in the acquiring section 110. Theaccessory information is then transferred to and stored in the DB 308together with the image data.

[0209] The user terminal 306 comprises a display section 324 that candisplay the image; an input section 326 that inputs the searchingcondition, image, symbol, language, or information for print order; andan interface (I/F) 328 that transfers the image, searching condition ororder information to the searching apparatus 302. The searching section314 in the searching apparatus 302 includes an I/F 315 for receiving animage or various kinds of information from the I/F 328 of the userterminal 306.

[0210] The acquiring section 310 in the searching apparatus 302comprises a tablet 330, an input subsection 332, a speech recognizingsubsection 334, an extracting subsection 336, and an accessoryinformation setting subsection (which is hereunder simply referred to assetting subsection) 338. The tablet 330 inputs freehand the outline ofthe specific information including specific geometric figures, specificsubjects, message information, personal information consisting of thefacial image of a specific person. The input subsection 332 inputs theoutline of the specific information including the specific geometricfigures, specific subjects, message information, personal informationconsisting of the facial image and the name of a specific person, andediting data of the CG image, and the meaning of the specificinformation comprising the audio information by means of GUI (graphicaluser interface) with the keyboard and the mouse, and optionallyrecognizes and inputs the meaning of the message information, or inputsthe photographing information that is relevant to the time when theimage was taken and the map information in order to extract the specificinformation that accompanies the image, as exemplified by the placewhere the image was taken, or the name of the subject, and the positionof the subject on the map, as well as the position information, mapinformation and camera information that are relevant to the time whenthe image was taken in order to derive the specific informationincluding the information on the distance of the screen having thephotographed image, directly from the camera used in taking the image,or by the operator's operation using the keyboard and the mouse. Thespeech recognizing subsection 334 inputs the audio information thataccompanies the image or the audio information of the specificinformation that is found in the image, recognizes the meaning thereof,and converts the thus recognized meaning to the specific informationrecognized by letters. The extracting subsection 336 uses the outline ofthe specific information input from the tablet 330, the input subsection332 and the speech recognizing subsection 334, and various kinds ofinformation at the time when the image was taken, to extract thespecific information from the image data of the image received from theimage processing section 320 of the photoprinter 304. The settingsubsection 338 sets from the specific information extracted in theextracting subsection 336, the specific information per se or therelated information, for example, the recognized specific informationincluding the message information and the audio information; the relatedinformation including the position, size, direction, color, presence orabsence and direction in which the specific information is moved(direction of its motion vector) in the case of an animation image forthe specific information including specific geometric figures, specificsubjects, and personal information consisting of the facial image andthe name of a specific person; the specific information per se includingthe editing data of the CD image; the specific information per seincluding the message information and audio information recognized inthe speech recognizing subsection 334 and the input subsection 332; thespecific information per se including the place where the image wastaken, or the name of the subject, and the position of the subject onthe map; and the specific information per se including the informationon the distance of the screen having the photographed image as theaccessory information to be stored in the DB 308. The thus set accessoryinformation is then transferred to the image processing section 320 inassociation with the image data. It should be noted here that the mapinformation may be previously stored in a database (not shown) within oroutside the extracting subsection 336 or in the DB 308 so that thestored information can be read as required.

[0211] Therefore, the image processing section 320 of the photoprinter304 transfers the image data of the image of interest together with theaccessory information transferred from the setting subsection 338 of theacquiring section 310 to the DB 308, where the image data and theaccessory information are as such stored. The image data to betransferred from the image processing section 320 to the extractingsubsection 336 and/or the DB 308 may be the one before image processingor the processed one. Both image data may be identical to or differentfrom each other. The image data that is not processed yet may betransferred from the image input section 318.

[0212] In the case of a CG image, editing data that is appendant to theimage data as the accessory information may be directly transferred fromthe CG image creating device 317 to the setting subsection 338.Alternatively, the image data and the appendant editing data may bedirectly transferred from the CG image creating device 317 to the DB308.

[0213] Instead of the photoprinter 304, another photoprinter, an imagereading apparatus (scanner), an image processing apparatus or anotherimage data supply source such as a driver of an image data recordingmedium may be used to receive the image data for acquiring the accessoryinformation in the acquiring section 310 and the image data for storingin the DB 308. Further, if two kinds of image data are derived from thesame image, one may be received from the photoprinter 304, the otherfrom another image data supply source.

[0214] The designating section 312 of the searching apparatus 302 usesthe tablet 330, the input subsection 332 and the speech recognizingsubsection 334 in the acquiring section 310 and the CG creating device317 to designate as the searching condition at least a portion of theaccessory information, for example the outline of the specificinformation per se such as the audio information and the messageinformation; at least one of the related information such as thepresence or absence of the specific information, position, size,direction, direction of its motion vector or color; and also designateas the reference data the information on the distance of the screen, forexample the position or name of the place where the image was taken orthe subject, the outline of the depth in each region of the screen, andthe outline of the depth obtained by positioning a patterned subject ina virtual map space. The tablet 330, the input subsection 332, thespeech recognizing subsection 334 and the CG creating device 317 used inthe designating section 312 may be the same as or different from thoseused in the acquiring section 310 or the like. Further, the acquiringsection 310 and the designating section 312 may not be separatelyprovided, but one component may be shared to acquire the accessoryinformation of the image data and designate at least one of theaccessory information as the searching condition.

[0215] As a result of searching in the searching section 314, the imagedata read from the DB 308 is displayed on the display section 316 as animage, but the displayed image need not be highly precise, if theselected image can be confirmed. Therefore, the image data read from theDB 308 may of course be displayed as it is, but highly precise image maybe displayed after processing has been performed in the image processingsection 320 of the photoprinter 304, if the display section 316 candisplay this image. That is, the display section 316 may be shared forthe searching apparatus 302 and the photoprinter 304. Further, in orderto confirm the image data selected in the searching section 314, theimage data may be directly transferred to the print output section 322of the photoprinter 304 so that a print-can be output therefrom.Alternatively, this image data may be transferred to another printer ormore simplified printer for print output.

[0216] In the image processing method of the invention, print reorder ispossible in the presence of the print (hard copy image) P, even if thereis no film F or image data recording medium R. If the specificinformation such as the specific figures and specific subjects has apixel density (resolution) required for researching, the image data maybe read from the print P with a scanner for reading a reflectionoriginal that is provided in the image input section 318 of thephotoprinter 304, or with another separate scanner or more simplifiedscanner. In the latter case, the thus read image data can be thendirectly transferred to the extracting subsection 336 of the acquiringsection 310. At that time, the specific information extracted in theextracting subsection 336 and/or the related information may be directlytransferred to the searching section 314 as such or after having beenconverted to the accessory information in the setting subsection 338. Inthis case, the image data read from the DB 308 is also subjected to theextraction of the specific information. Then, the image data read fromthe DB 308 to the searching section 314 may be directly transferred tothe extracting subsection 336 of the acquiring section 310.

[0217] In addition, when the degree of agreement of the specificinformation such as the specific figures and specific subjects in aplurality of images for which an order was made is calculated in thesearching section 314, the result of extraction of the specificinformation in the extracting subsection 336 is preferably transferredto the searching section 314 directly.

[0218] The foregoing embodiments are given for illustrative purposesonly and it should be understood that the invention is by no meanslimited to those embodiments and various improvements and designmodifications are of course possible without departing from the spiritand scope of the invention.

[0219] As described above in detail, according to the first embodimentof the present invention, there can be realized an image searchingmethod capable of efficient image search using as a key the shapes (e.g.specific geometric figures) of subjects (human individuals and physicalobjects) that are found in the original image, the specific subjects perse, or message information and audio information that were synthesizedinto or accompany the original image and, in addition, an imageprocessing method that utilizes the image searching method or thespecific information extracted from the original image can be realizedaccording to the third embodiment of the invention.

[0220] Stated more specifically, the image searching method of theinvention offers the first advantage that an image complying with thegiven instruction can be searched for by a simple way of givinginstructions such as drawing a rough sketch of figures on a tablet or animage display device, inputting the names or symbols of figures from akeyboard or inputting these instructions by speech. Even if a person isto be searched for, an image that complies with the given instructioncan be searched for by a simple way of giving instructions such asdrawing on the tablet a rough sketch of the person per se or at leastone element selected from among his or her position, size and directionor inputting from the keyboard the presence or absence of the person orat least one element selected from among his or her position, size anddirection.

[0221] Secondly, if a person in a dynamically changing image is to besearched for, a dynamically changing image that complies with the giveninstruction can be searched for by a simple way of giving instructionssuch as roughly showing its initial position and the direction of itsmovement on the tablet or inputting them from the keyboard or by speech.As a result, when editing videos, images can be easily searched throughby rough scene description.

[0222] Thirdly, if the images to be searched through have audioinformation, it may be used to search for an image that complies withthe given instruction. Since dynamically changing images as well asstatic images can be searched through, search can be made not only fromthe movement of the subject but also from the contents of his or herspeech, offering great benefits to the case of editing videos.

[0223] Fourthly, the facial image of the person to be searched for andhis or her name are keyed to each other when they are transformed toaccessory information and an image that complies with the giveninstruction can be searched for by making use of that information. Forexample, a photo full of memory can be easily searched for by using thename of a family member as a keyword. The images to be searched throughinclude not only static images but also dynamically changing images, aswell as images having audio information.

[0224] Fifthly, when searching through images having audio information,various kinds of audio information may be combined with each other.Again, the images to be searched through include not only static imagesbut also dynamically changing images.

[0225] Sixthly, if the data for the creation and editing of the CGimages to be searched through is used as accessory information, thevolume of calculations that must be made in the process of searchingthrough CG images can be sufficiently reduced to ensure rapid andprecise searching for a CG image that complies with the userinstruction. Needless to say, the CG images to be searched through maybe static or dynamically changing. If dynamically changing CG images areto be searched through, specific figures or objects and their motionvectors (representing either their size or direction or both) arepreferably designated as described above. When searching through imageshaving audio information, various kinds of audio information may becombined with each other.

[0226] Seventhly, if text data such as letters or reference image dataare used as message information that accompanies the images to besearched through, an image that complies with the given instruction canbe searched for by a simple way of designation such as designating thecharacteristic values of letters per se, their colors, contents and theshape of handwriting or by another simple way of designation such asdesignating a reference image area from a plurality of preliminarilyregistered reference images or selecting one of the prepared pictorialpatterns.

[0227] If the user wants to display an image as the result of searching,the area that complies with the designated searching condition(s),excepting motionless parts such as the background, may be displayed inpreference to other areas. In another effective way, the area complyingwith the designated searching condition(s) may be displayed as enhancedover the other areas after it has been subjected to special imageprocessing such as luminance enhancement or blinking.

[0228] When the desired image is to be displayed, it is preferred tocheck the performance of the display device and let it display the imageafter the image data has been processed to meet the specific performanceof the display device. According to another effective approach, theareas that have been searched through are managed on the database areaby area and the priority of reference in search mode or the imageresolution for data storage is set in accordance with access frequency.

[0229] By applying the above-described image searching method, the imageprocessing conditions adopted in the previous printing can be searchedfrom the image submitted to the service bureau at the time of laterprinting and prints are produced that have high enough fidelity tosatisfy the customer.

[0230] In an application of this image searching method, similar imagesare differentiated on the basis of their compositions, so itsperformance is less sensitive to the effects of color and density; as aresult, there is no possibility that prints having extremely differentcolors and densities are produced from photographic films or capturedimage recording media of a single order and no customer will leave theservice bureau dissatisfied.

[0231] As described above in detail, the present invention also offersthe advantage of realizing an image searching method that is applicablewhen the customer wants to locate a desired image from a number ofstored images and which enables easy image search by utilizing theposition information, camera information and map information that arerelevant to the time when the image was taken and which is added inassociation with the recorded image.

[0232] Thus, according to the image searching method of the secondembodiment of the invention, an image that complies with the giveninstruction can be searched for on the basis of a designated searchingpattern the user has input, typically with a mouse, on the screen of aterminal device or a like image display device or on the basis of theposition information (latitude and longitude) about the subjects in thescreen, their names (e.g. place name) and the like that have been inputas searching information from a keyboard.

[0233] If the user adds a 2D layout of subjects to the input searchingpattern, an image that complies with the given instruction can besearched for much more easily.

[0234] Remember that in principle the image searching method of theinvention is applicable not only to static images but also todynamically changing images.

1-35. (Cancelled)
 36. An image processing method; comprising the stepsof: extracting specific figures or specific subjects from a plurality ofimages within a single order; and performing image processing of imageframes to yield prints that are finished to have quality within apredetermined range if said image frames are similar in terms ofpositions and sizes of said specific figures or subjects.